Carl Franzen writes that DeepSeek has launched V4.1-Flash, a model featuring a 552-billion-parameter mixture-of-experts backbone designed to drastically reduce costs for long-context workflows through specialized caching and architecture. The model offers extremely low off-peak rates of $0.003 per million cached input tokens, making it highly competitive against frontier models like GPT-5.6 Sol and Claude Opus 5 when used in repetitive agentic loops. While its total parameter count has increased significantly compared to previous versions, its Causal Encoder-Decoder architecture aims to minimize compute requirements during the prefill stage of inference.
- V4.1-Flash utilizes a Causal Encoder-Decoder architecture that activates 8 billion parameters during prefill and 16 billion during decoding.
- The model features an extremely high context window of up to 1 million tokens.
- DeepSeek's technical report highlights the use of FP4 KV caching, which reduces global KV cache size by approximately one-quarter compared to its predecessor.
- Off-peak hours for lower pricing are scheduled from Monday through Friday, specifically between 01:00–04:00 UTC and 06:00–10:00 UTC.
- V4.1-Flash utilizes a Causal Encoder-Decoder architecture that activates 8 billion parameters during prefill and 16 billion during decoding.
- The model features an extremely high context window of up to 1 million tokens.
- DeepSeek's technical report highlights the use of FP4 KV caching, which reduces global KV cache size by approximately one-quarter compared to its predecessor.
- Off-peak hours for lower pricing are scheduled from Monday through Friday, specifically between 01:00–04:00 UTC and 06:00–10:00 UTC.
noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
Deploying Large Language Models in streaming applications is limited by growing KV cache memory during decoding and an inability to generalize beyond training sequence length, with naive window attention failing once text exceeds the cache size. The authors observe an attention sink phenomenon where models assign strong attention scores to initial tokens even when semantically irrelevant, and find that retaining the KV of those initial sinks together with a sliding window recovers performance. This motivates StreamingLLM, a zero-shot framework that enables LLMs trained on finite windows to generalize to infinite sequence lengths without fine-tuning, achieving stable language modeling up to 4 million tokens on Llama-2, MPT, Falcon and Pythia.
- Attention sinks arise from strong attention to initial tokens acting as a sink for excess attention mass.
- A placeholder token added as a dedicated attention sink during pre-training further improves streaming deployment.
- StreamingLLM achieves up to 22.2x speedup over sliding-window recomputation in streaming settings.
- Paper is ICLR 2024 and code/datasets are released at mit-han-lab/streaming-llm.
- Attention sinks arise from strong attention to initial tokens acting as a sink for excess attention mass.
- A placeholder token added as a dedicated attention sink during pre-training further improves streaming deployment.
- StreamingLLM achieves up to 22.2x speedup over sliding-window recomputation in streaming settings.
- Paper is ICLR 2024 and code/datasets are released at mit-han-lab/streaming-llm.
Google’s TurboQuant Compression May Support Faster Inference, Same Accuracy on Less Capable HardwareGoogle Research has introduced TurboQuant, a new quantization algorithm designed to compress the Key-Value (KV) cache of large language models by up to 6x. By utilizing a two-step process involving randomized Hadamard transforms and Quantized Johnson-Lindenstrauss transforms, the method achieves 3.5-bit compression with near-zero accuracy loss on benchmarks like LongBench. This optimization addresses the massive VRAM requirements of long-context windows, potentially allowing large models to run on significantly less powerful hardware.
Key points:
* Compresses KV cache down to 3.5 bits per value.
* Maintains inference accuracy without requiring model retraining.
* Uses data vector rotation and QJL transforms to handle outlier distribution skew.
* Reduces the memory bottleneck for long-context LLM inference.
* Enables massive context windows on more modest hardware configurations.
Key points:
* Compresses KV cache down to 3.5 bits per value.
* Maintains inference accuracy without requiring model retraining.
* Uses data vector rotation and QJL transforms to handle outlier distribution skew.
* Reduces the memory bottleneck for long-context LLM inference.
* Enables massive context windows on more modest hardware configurations.
This article explores TurboQuant, a new vector quantization method introduced by Google researchers to address the massive memory requirements of Large Language Models (LLMs). As LLM parameters and Key-Value (KV) caches grow, memory management becomes a critical bottleneck for performance. TurboQuant utilizes the PolarQuant algorithm and the quantized Johnson-Lindenstrauss (QJL) algorithm to compress the KV cache significantly. Google claims this method can achieve up to 6x compression levels without a noticeable impact on inference times or accuracy. While the article notes that Google's benchmarking data is somewhat vague compared to competitors like NVIDIA's NVFP4, TurboQuant represents a significant development in optimizing AI hardware compatibility and real-time inference performance.
This paper introduces KVTC, a lightweight transform coder designed to compress key-value (KV) caches, which are crucial for efficient large language model (LLM) serving. KV caches enable reuse across conversation turns, but can consume significant GPU memory. KVTC addresses this by applying techniques from classical media compression – PCA-based decorrelation, adaptive quantization, and entropy coding – to reduce cache size without requiring changes to the underlying model. The authors demonstrate that KVTC achieves up to 20x compression while maintaining reasoning accuracy and long-context performance, and even higher compression for specific applications.
A deep dive into the process of LLM inference, covering tokenization, transformer architecture, KV caching, and optimization techniques for efficient text generation.
K8S-native cluster-wide deployment for vLLM. Provides a reference implementation for building an inference stack on top of vLLM, enabling scaling, monitoring, request routing, and KV cache offloading with easy cloud deployment.
vLLM Production Stack provides a reference implementation on how to build an inference stack on top of vLLM, allowing for scalable, monitored, and performant LLM deployments using Kubernetes and Helm.
The article discusses how the Key-Value (KV) Cache is used to optimize the inference process of Large Language Models (LLMs) by reducing redundant computations and improving performance.
